1. Field of the Invention
This invention relates to a device, system, and method for electronically capturing hand-written notes and sketches.
The invention also relates to a cradle arranged to be clipped onto or otherwise removably secured to a writing instrument, and to hold a camera or other optical image capture device (such as an optical image scanner or a lens connected to a remote camera by fiber optics, hereinafter referred to as an optical reader).
In addition, the invention relates to a note or sketch transcription device that includes an optical reader or camera held by a cradle that may be clipped or otherwise removably secured to a writing instrument, and means such as an electrical cable, fiber optic cable, or wireless transmitter for transmitting images of hand-written notes from the optical reader or camera to a central processing unit as they are being written.
Furthermore, the invention relates to a system that includes an optical reader or camera, a cradle for removably clipping or otherwise securing the optical reader or camera to a writing instrument, means such as an electrical cable, fiber optic cable, or wireless transmitter for transmitting images of hand-written notes from the optical reader or camera to a central processing unit as they are being written, and a central processing unit for receiving, processing, viewing, and/or storing the optical images.
Finally, the invention relates to a hand-written note or sketch transcription method that includes the steps of removably clipping an optical reader or camera onto a writing instrument, taking notes or drawing sketches using the writing instrument while transmitting images of the notes or sketches from the optical reader or camera back to the central processing unit for storage, display, and/or analysis, removing the cradle from the writing instrument, and removably clipping the cradle onto another writing instrument.
The device, system, and method of the invention are intended to be used with any type of hand-held writing instrument including, but not limited to, pens, lead pencils, chalk, grease pencils, and markers. Among the applications of the device, system, and method of the invention are transcription of materials written on a chalkboard by teachers or coaches, notes taken by students or employees at lectures or meetings, homework assignments by a bed-ridden or handicapped student, crime scene and suspect descriptions or sketches taken by police officers, orders taken by servers in restaurants, prescriptions written by doctors, hand-written Chinese or Japanese ideograms, and hand-written letters in languages requiring accent or inflection marks not part of a regular alphabet, such as Vietnamese or Arabic.
A significant difference between the device, system, and method of the invention and prior devices, systems, and methods for capturing images of hand-written notes, messages, or signatures, lies in the use of the afore-mentioned cradle. The cradle permits the optical reader or camera to be retro-fitted onto any of a variety of existing writing instruments, resulting in far greater versatility than is possible with prior handwriting-transcription devices, systems, and methods.
2. Description of Related Art
Despite the widespread availability of hand-held computers, personal digital assistants, and other portable data entry devices, traditional non-electronic writing instruments such as pens and pencils remain the preferred means of taking notes and creating sketches. Such writing instruments are inexpensive, convenient to carry, extremely easy to use, and essential for spontaneous presentations such as writing of mathematical formulas during a lecture, diagraming of football or basketball plays on a clipboard, and so forth. In fact, the human hand, eye, and brain are uniquely adapted to manipulating traditional writing instruments, whereas keyboards and other computer input devices tend to strain muscles, tendons, nerves, and eyes and, in case of frequent use, can cause severe and even crippling injuries due to the un-natural motions required.
The drawback of traditional writing instruments is that there is no convenient way to enable viewing of the hand-written notes, sketches, diagrams, and the like by persons remote from the location of the writer, or of saving the notes in a form that facilitates later viewing, editing, or distribution. To accomplish such remote viewing or saving of hand-written notes, the notes must be captured electronically, but electronic capture requires laborious manual input of the notes into a computing device, or time-consuming scanning of each page of notes, and is essentially impossible when the notes are written on an impermanent medium such as a chalkboard.
It has previously been proposed to eliminate the need for manual data entry of scanning by capturing the handwritten notes or sketches as they are written, using an appropriately positioned scanner or camera. The previously proposed devices, systems, and methods most relevant to the present invention have generally fallen into three categories:                I. optical readers or cameras integrated with specially-designed writing instruments;        II. systems for capturing images of entire chalkboards and other varied writing surfaces; and        III. non-optical sensors designed to be mounted on conventional writing instruments.        
The basic principle of integrating an optical reader or video camera with a writing instrument has been known at least since 1965, when U.S. Pat. No. 3,182,295 issued. As concisely stated therein, integration of the optical reader and writing instrument provides a convenient “means for indicating instantaneously at remote points whatever message is being written with the utensil.” The present invention, and the devices disclosed in such later patents as U.S. Pat. Nos. 4,475,240; 5,960,124; 5,774,602; 5,748,808; 5,215,397; 5,107,541 all share this basic principle. However, despite significant advances in scanner and camera technology since 1965, integrated optical reader/writing instrument devices have yet tp attain widespread acceptance. The reason is that the prior integrated devices all possess the disadvantages, which are overcome by the present invention, that they can only be used with a single type of writing instrument, and that when the writing instrument is worn out or broken, the optical reader is rendered useless. Most individuals become accustomed-to and favor particular types of writing instruments, based on the shape of the barrel, quality of ink, lead, or other writing medium, and overall appearance of the instrument, and it is very difficult and costly to produce integrated writing/optical reader instruments that correspond in type to all of the different writing instruments currently in use. Use of the optical reader is more likely if the user does not have to give up a favored writing instrument, or abandon existing inventories of such instruments.
So far, the only way to enable transcription of notes irrespective of the type of writing instrument used to take the notes, has been to simply capture images of the entire surface on which the notes or sketches are written, as proposed in U.S. Pat. Nos. 6,044,165 and 4,511,929. These patents explicitly recognize the need to capture, for example, “handwriting on a whiteboard or chalkboard,” but reject the possibility of mounting the camera on a chalk or whiteboard marker. While solving the problem of restriction to a particular writing instrument, however, such image capture systems and methods require bulky equipment and are difficult and time consuming to set up, making them impractical for most note-taking applications.
The third category of previously-proposed hand-writing capture devices, systems, and methods, involving non-optical motion sensors that may be retrofitted onto conventional writing instruments, provides greater convenience and flexibility, but in general can only be used for the more limited purpose of capturing features or characteristics of the handwriting rather than its content, and are primarily intended for signature verification. Examples include motion detectors designed to be clipped onto writing instruments, as disclosed in U.S. Pat. No. 5,781,661; a stylus housing containing a transmit portion signals to a set of receivers, the stylus housing being designed to house a variety of writing elements including chalk, pencil lead, and markers as disclosed in U.S. Pat. No. 6,111,565; and a biometrics sensor provided in the form of a “stylus grip 30 . . . that is compatible with conventional styluses and pencils,” as disclosed in U.S. Pat. No. 6,307,956. The latter patent is of particular interest because the biometrics sensing “stylus grip” may be used with a “stylus” that includes an optical or imaging print sensor, but there is no suggestion in the patent to enable the grip itself to hold the optical or imaging print sensor so that the print sensor rather than the biometrics sensor may be removably secured to different writing instruments.
Also of interest as background with respect to the present invention are U.S. Pat. No. 4,856,077, which discloses an integrated optical reader/writing instrument designed to capture the crossing points of a signature rather than an image of the entire signature or other hand-written markings, U.S. Pat. No. 5,247,170, which discloses an optical reader integrated with a highlighter pen but which is designed to read the text being highlighted rather than the marks made by the highlighter pen; and U.S. Pat. No. 5,939,702, which discloses a “writing instrument having an integrated optical reader” that clearly is not designed to read text or images written by the writing instrument, as evidenced by the fact that the writing tip and reader tip are never made accessible at the same time (col. 3, lines 29–36), the reader tip being provided instead for general purpose scanning.
Finally by way of general background, U.S. Pat. Nos. 5,548,092; 5,434,371; 5,247,137; and 5,159,321 are of interest for their disclosures of writing instruments in which marks made by the writing instrument are recorded based on signals from integral, as opposed to removable, strain gauges, pressure sensors, accelerometers, and/or other types of non-optical motion or force sensors, while U.S. Pat. Nos. 6,310,988; 5,745,177; 5,301,243; and 5,574,804 disclose pen-shaped scanners and video cameras that do not include any sort of writing instrument and which therefore cannot be used to capture images of hand-written notes or sketches to be captured as they are written.